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ABSTRACT 

The applicability is explored of the Bayesian 
random-affect analysis of variance (ANOVA) model developed by G. C. 
Tiao and W. Y. Tan (1966) and a method suggested by H. K. Suen and P. 
S. Lee (1987) for the generalizability analysis of autocorrelated 
data. According to Tiao and Tan, if time series data could be 
described as a first-order autoregressive series with parameter "p" 
(rho) , unbiased estimates of random error variance could be derived 
via a Bayosian process. Suen and Lee's two-step alternative procedure 
combines both Box-Jenkins time series analysis and a random-effect 
ANOVA process. The autocorrelated component of the data can be 
removed through the Box- Jenkins procedure, and then the residual or 
white-noise data can by analyzed via the ANOVA process to produce 
unbiased variance estimates. Theoretical advantages and limitations 
of the two approaches are outlined, focusing on autoregressive 
integrated moving averages. Three analyses of the methods are 
presented. Results from application of the methods to numerous data 
sets show that autocorrelation has a negligible or no effect on the 
systematic variance across observers. The Suen-Lee method is superior 
to the Tiao-Tan method in applications to generalizability assessment 
of observation data. Based on 28 behavioral observation time series, 
the Suen-Lee method seems to be applicable only when the relative 
systematic observer variance is small. A 29-item list of references 
and one data table are provided. (TJH) 
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INTRODUCTION 

The purpose o-f this paper is to explore the applicability o-f the Bayesian 
random-s-f-fect ANOVA model developed by Tiao and Tan <1966) and a method 
suggested by Suen and Lee <I987a, 1987b) for the general i zabi 1 i ty analysis o-f 
autocorrelated data. 

The general izabi 1 i ty theory o-f measurement (Cronbach et a1 ., 1972, 
Brennan, 1983) has been suggested by many as particularly suited -for the 
assessment o-f data acquired through the direct observation o-f behavior (e.g., 
Bakeman & Gottman, 1986; Berk, 1979; Cone, 1978; Hartmann, 1982; Kazdin, 1977; 
Mitchell, 1979), a method commonly used in classroom research, special 
education, and clinical and counseling psychology. A major obstacle to this 
application is the presence o-f autocorrelation o-f ten -found among direct 
observation time-teries data <c-f. Gardner et al . , 1982; Glass et 1975] 
Gottman & Glass, 1978; Hartmann et al., 1980; Jones et al . , 1977, 1978). It is 
generally recognized that in the presence o-f correlated errors, estimates o-f 
expected mean squares (Box, 1954), expected variance ccfTiponents (Brennan, 
1984), and subsequently reliability indices such as KR-20 (Maxwell, 1968) are 
biased, potentially leading to -false indications o-f data reliability or a lack 
thereo-f . 

The extent to which dat'* acquired through the direct observation o-f 
behavior are autocorrelated is currently unclear and controversial. On the one 
hand, while authors such as Gardner et al . (1982), Glass et al . , (1975); 
Gottman & Glass (1978), and Hartmann et al . (1980) suggc^vted or deduced that 
many behavioral data are quite likely to be autocorrelated, Jones et al . (1977, 
1978) produced empirical evidence that the majority o-f published behavioral 
data are in -fact autocorrelated. This conclusion was severely challenged by 
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Huitema (1985), who reanalyzed Jones et al.'s data as well as other published 
data and concluded that the overwhelming majority o-f data acquired through 
direct observations o-f behavior are not autocorrelated. Huitema's conclusion 
o-f no autocorrelation was, in turn, challenged by others on methodological 
grounds (e.g., Suen, 1987; Suen & Ary, 1987; Sharpley & Alavosius, in press; 
Marascuillo & Busk, in press). Currently, the only conclusion one can draw is 
that it is inconclusive as to whether autocorrelation is a prevailing 
characteristic o-f observational data. 

Although it is inconclusive as to the extent to which observational dsta 
are autocorrelated, everyday experiences suggest that at least some o-f the 
behavioral data are autocorrelated. When data are acquired by observing the 
occurrences o-f a behavior over time, it is reasonable to assume that, at least 
in some cases, the extent o-f occurrence o-f the behavior at a point in time is 
in-fluenced by the extent o-f occurrence o-f the same behavior at previous points 
in time. 

In the presence o-f autocorrelation, to produce unbiased estimates o-f 
variance components, which is an essential step in general izabi 1 i ty analysis 
(i.e., for both G and D studies), at least two methods have been developed. 
Tiao and Tar. '.1966^ developed a Bayesian process -for a one-way ANOVA which can 
be used when the nature o-f the autocorrelation can be described as a 
•first-order autoregressi ve model (i.e., AR(1)). This method can theoretically 
be extended to an n-way ANOVA as v^ell as higher order au toregressivre models. 
Suen and Lee (1987a, 1987b), on the other hand, developed a two-step process by 
combining the Box-Jenkins (Box tc Jenkins, 1976) autoregressi ve integrated 
moving average (ARIMA) modelling process with ANOVA. There are some 
theoretical advantages and disadvantages to each o-f these two methods. 
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Hov'iever, possible applied problems associated with th.^se two methods have not 
been explored. The purpose o-f this paper^ is fo explore these possible 
probi ems. 



THE TIAO-TAN METHOD 
Tiao and Tan <1966) suggested, i-f the data in a time series can be 
described as a Ist-order autoregressi ve series with parameter P, unbiased 
estimates o-f random error variance can be derived through a Bayesian process. 
For a k-observer by n-points o-f observation matrix, assuming a uni-form prior 
distribution (i.e., no prior i n-f ormat i on) , the posterior distribution o-f random 
error variance is reflected by the posterior distribution o-f V=l/0^^ which 
is: 



p<V!p,y) = Sj(p)^^{k(m-1)} . <1) 

H^(p){«<k-l),>5k<m-l)} 

Inhere y is the data vector o-f p'i observer, G is the Gamma -function at 
>5*(p)t with parameter ^(k-1), m is n-1 , and Sj(p) is de-fined as: 



Sj(P) = bj+Cj(p-pj)^, 



k m-H km 
i=l j=2 ' ^ i = l j=l ''^ ' 
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1 m+1 
— S 

m j=:2 



1 m 
m j = l 



km 
^ i=l j=l 'J ' 



Additionally, ^or Eq . (1), ^^{k<m-l ) > is the density at t a ■/ ^unctic 
with k<m-I) degrees o-f -freedom where t=Sj(p)/<j'. The <J(P) value -for the 
Gamma -function in Eq . (1) is de-tined as *<P)=S2<p)/Sj <p) , where 



S2<P) = b2+C2<P-p2)^» 



b2 = m 2 ,y<*^-7*h'-m$' 2 (;.<->-7-^^ 

i=l ' i=l ' 



k _ 

= m 2 <y/"'-y^ , 
i = l ' 



1 K _ 

V . 
I 



k i = l 
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1 k _ 
and y^"^ = s y/"V 



Finally, -for Eq. (1) : 



with the right side o-f the equation being the beta -function at <Kp)/(l+(Kp)) 
with parameters and ^k(m-l). With the known posterior distribution o-f 

error variance as de-fined in Eq . (1), unbiased estimates o-f variance components 
can be obtained by identi-fying the modal maximum likelihood variance o-f the 
posterior distribution. 



THE SUEN-LEE ALTERhWTiVE 



Suen and Lee (1987a) proposed a two-step alternative procedure which combines 
both Box-Jenkins time series analysis (Box & Jenkins, 1976) and the random 
e-f-fect ANOVA process. First, the autocorrel ated component o-f the data can be 
removed through the Box-Jenkins procedure and then the residual or white-noise 
data can be analyzed through the ANOVA process to ^^roduce unbiased variance 
estimates. For a T-time by K-observer single— facet crossed-design random 
e-f-fect general izabi 1 i ty analysis with serial dependency along T, the time 
dimension, the total variance can be decomposed as -follows: 
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2 . . 2 



<2) 



I on 



where (0^ ^nd (0^^ are **moderat i ng" e-f-fects on the T and the interacti( 
variances due to autocorrelation. Both (0^ and (0^^ are negative in value 
because it was deduced that autocorrelation should lead to underestimates o-f 
the variance -for T and the random error variance. This can be veri-fied -from 
the characteristic o-f t-ratios. In the presence o-f autocorrelation, it is 
commonly known <e.g., McDowall et al.j 1978; Sharpley & Alavosius, in press) 
that the t-ratio is spuriously in-flated. It is al50 known that a mean is a 
deterministic statistic and is not in-fluenced by the presence o-f autocorrela- 
tion. Hence, the only explanation -for the in-flation o-f t is that the standard 
error is underestimated* In othe'^ words, the variance is underestimated. 

Suen & Lee suggested that unbiased estimates o-f variance components can be 
obtained by applying the Box-Jenkins procedure (Box & Jenkins, 1976) on the 
vector o-f mean scores across observers to identi-fy the best ARIMA<p,d,q) model. 
The general ARIMA model ^or the mean, Mj , the i th point in time is; 



= ^i^S^i-I^*2^i-2^--Vi-p'^l^-r^2f^i-2^--'^^-q' 



(3) 



where E. js the random or white noise component at time i, is the pth 
order au toregressi ve parameter, and 6^ is the qth order moving average para- 
meter. The identi-fied model can then be applied to the data o-f each observer 
individually to "-filter" autocorrelation -from the data. Spec i -f ical 1y, the 
score -from the i th observation o-f the j th observer can be "-filtered" through: 



^ ij ^ij"*l^'<i-l)j"---"'*p^'<i-p)j"^l^<i-l)j-----V<i-q)j' 



<4> 
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Where X'-^ jg ^^e "filtered" and Xjj is the raw score at the i th point in 
time -for the jth obseri/er. 

An ANOVA o-f the "-filtered" data matrix with elements X^.. would produce 
unbiased variance estimates. The (0 values in Eq. <2) can then be obtained by 
contrasting the di-f-ferences o-f the results o-f ANOVA^s on the "-filtered" and 
"un-filtered" data. A problem with Eq. <4) is that Xj^. has to be assumed to 
contain only a random component since 5<^j^pjj is not available. This is 
counter-intuitive. To resolve this problem, Suen i Lee (1987b) later suggested 
the use o-f bacK-f orecast i ng techniques (Box & Jenkins, 1976). 



THEORETICAL ADVANTAGES AND LIMITATIONS OF THE TWO METHODS 



For the purpose o-f general izabi 1 i ty assessment, a -fundamental problem with 
the Suen-Lee proposal is that, through the ARIMA process, the random errors o-f 
variance estimates (i.e., variances o-f variance estimates, VAR(0^^), 
VAR($|^^), & VAR(0^|^^)) are expected to be inflated by the error o^ 
speci-f ication o-f the ARIMA model. Spec i -f i call y, because the ARIMA analysis is 
performed on the mean score series, not the data streams o-f individual 
observers, anJ ^et individual observer series were -filtered based on the model 
•for the mean series, errors in speci-fying the model -for the mean series as the 
representative series -for all observers will compound the random error o-f the 
variance estimates. Suen & Lee^s argument was that i-f the in-flation o-f these 
variances o-f variance estimates is small, the process is e-f-fective. I-f they 
prove to be large, systematic errors are replaced by random errors. The exact 
amount o-f in-flation, however, is unknown. The major advantage o-f the Suen-Lee 
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method is that it can b<> applied to any autocorre Jated data regardless of the 
nature o-f the autocorrelation. That is, the Suen-Lee method can be applied to 
any ARIMA model as well as cyclical SARIMA model. 

The Tiao-Tan method, on the other hand, offers several advantages over the 
Suen-Lee procedure. First, instead of being an ARIMA method, the Tiao-Tan 
method is a randcm effects model and is hence consistent with the general i- 
zability theory. Second, the variances of the final variance components can be 
derived. Finally, the inflation of these variances of variances can be 
expected to be smaller than those with the Suen-Lee method. 

On the other hand, the Tiao-Tan method contains some limitations. First, 
fis a Bayesian process, some assumptioris have to be made regarding the form of 
the prior distribution. Tiao and Tan pointed out that different likelihood 
functions, and subsequently posterior distributions of variance estimates, will 
result if a normal prior distribution rather than a uniform prior distribution 
is assumed. Second, the method is appropriate only if the data form a 
Ist-order au toregressi ve series. Although Tiao and Tan suggested that their 
method can be extended to higher-order autoregressi ve series, it is 
inappropriate for moving average series ARIMA<0 ,0 ,q) , mixed series 
ARIMA<p ,0,q) , and seasonal or cyclical series SARIMA<p ,d,q) . Finally, and 
perhaps most seriously from an applied perspective, the Tiao-Tan method 
requires derivations of complex Gamma and Beta f'Jnctions. The complexity is 
particularly severe with the Gamma function. Since computer softwares are not 
available, the use of the Tiao-Tan method also depends on the ability to 
develop and compile a complex computer algorithm. 

Both the Tiao-Tan method and the Suen-Lee method are appropriate only for 
a single-facet random-effect ful ly-crossed-design general i zabi 1 i ty assessment. 
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While the Suen-Lee method \s specifically designed for th** random model 
involving a two-way ANOVA, the Tiao-Taii methca is designed for a one-way ANOVA, 



UhkiiNSUERED QUESTIONS 



With the limitations of the two methods, three major questions remain as 
to the applicability of the Tiao-Tan method and the Suen-Lee method for 
general izabi 1 i ty asse:>smeit. First, the Tiao-Tan method appears to offer some 
theoretical advantages over the Suen-Lee metijod, even considering the complex 
Gamma function for the former. Since the Tiao-Tan method is applicable only to 
ARIMA(1,0,0> series, the critical question \z how prevalent are ARIMA<1,0,0> 
series in behavioral observation data? If ARIMA<1,0,0> series are prevalent, 
Tiao-Tan appears to be the method of choice for general i zabi 1 i ty assessments of 
autocorrelated behavioral observation data. Considerations, should then be 
givpn to the development of appropriate software for the use of the method. 

If ARIMA<1,0,0> series is rare, the Suen-Lee method would be more useful 
since it applies to all forms of ARIMA<p,d,q> as well as SARIMA<p ,d.q) models. 
The question then is whether the Suen-Lee method can be refine < so that the 
expected inflation of the variances of variances due to misrepresentation by 
the mean series can be minimized? If it can be refined, what applied problems 
may arise with the refined approach? 

To answer this series of questions, three analyses were conducted as 
below. 
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FIRST ANALYSIS 

Given the current uncertainty regarding the extent to which behavioral 
data are autocorrelated (see earlier discussion), existing literature will not 
offer an answer to the first question of whether ARIMA<1,0,0) series are 
prevalent^ Nor would a secondary analysis of published data offer an answer 
since most observational data in the literature contain extremely sho. t ser .'S 
^cf. Huitema, 1985; Suen & Ary, 1937). Herce, a sample of primary data is used 
to assess the extent to which obsen'a U onal data contain ARIMA<1,0,0) series. 

SAMPLE DATA 

The raw observational data in Ary et al.'s <1986) study were used. Ary et 
al. employed 7 observers to record 4 behaviors of a child at 5-second intervals 
for 1 hour, generating a total of 720 observations per observer per behavior, 
or a total of 20,160 observation data points. Each data point is dichotomous 
(occur vs. no^ occur). For our purpose, each of the 72C -point dic'-itomous 
series was collapsed into a 144-point time series of prevalence scores (or 
proportion of time behavior occurs) by transforming dichotomous scores in every 
5 5-second intervals (representing 25 second real time) into a prevalence 
proportion (or p or mean score). Subsequently, a total of 28 144-point time 
series representing the preva'^nce of ) behaviors of a child as recorded by 7 
independent observers were obtained. 

DATA ANALYSIS 

To investigate the extent to which ARIMA(1,0,0) serie:; exists in 
behavioral data, the 28 time-series were submitted to 28 separate ARIMA 
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analyses. Specifically, analyses were performed through the ARIMA routine in 
MINITAB. In addition to estimating ARIMA parameters through the judgments of 
autocorrelation function (ACF) and partial autocorrelation function (PACF) 
correl ograms, the t-ratios of the estimated ARIMA parameters were examined to 
determine the significance of the parameter estimates. 



RESULTS AND CONCLUSION 

Of the 28 time series, none yielded a statistically significant ist-order 
autoregressi ve (ARI) parameter. One of the 28 series yielded statistically 
significant ARIMAd, 0,1) parameters (0^3,^774^ t=3.30, p<.05; ej=.655, 
t=2.36, '•05) and four yielded statistically significant seasonal 
autoregressive (i.e., SARIMA(9 ,0 ,0) ) parameters, with cycles of 9 data points 
= .217, .376, .363, & .266; t = 2.44, 4.37, 4.08, & 3.03; p<.05). In other 
words, of a sample of 28 behavioral observation time series, 5 (i.e., 18>i) 
showed some form of autocorrelation. However, none of these autocorrel ated 
data can be described as an ARIMAd, 0,0) series. 

To the extent that the limited sample of 28 data sets can represent 
typical behavioral observation data, a conclusion can be drawn that autocorre- 
lation does exist in a substantial portion (about I out of 5) of observational 
data. At the same time, an ARIMAd, 0,0) series is rare (none at all in this 
sample). Hence, the Tiao-Tan method may have only very limited utility for 
behavioral observation data. 



SECOND Ahk!^LYSIS 



The lack of ARIMAd, 0,0) series in behavioral observation data is not a 
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problem with the Suen-Lee method, which can be applied to all ARIMA and SARIMA 
series. One way to minimize the expected in-flation in the variances o^ 
variance estimates is to limit the application o-f the Suen-Lee method to 
situations in which the ARIMA or SARIMA models and parameters -for di-f-ferent 
observers o-f the same behavior are similar. In such a case, using the ARIMA or 
SARIMA model o-f the mean series as the representative model -for the data -from 
all observers wi 1 1 contain a minimal amount o-f misrepresentation. When this 
occurs, the in-flation o-f the variances o-f variance estimates can be expected to 
be minimal or negligible. 

For the purpose o-f the present analysis, two criteria were used to judge 
i-f the k series -from K observers o-f the same behavior are su-f-f iciently similar 
to be represented by the ARIMA parameters o-f the mean s^'^ies. First, the 
series -for individual observers need to -fit the same ARIMA<p,d,q) or 
SARIMA(p ,d,q) model. Second, the values o-f the K parameter estimates <i.e., 

A A 

♦ 's and 9^s as appropriate) need to be within one standard error o-f estimate o-f 
one another (i.e., middle 68X interval). The question is how o-ften can we 
expect di-f-ferent observers recording the same autocorrel ated behavior to yield 
data that meet these two criteria? 

DATA ANALYSIS AND RESULTS 

The results o-f the ARIMA analysis o-f the 5 ^>ets o-f autocorrel ated data 
•from Ary et al.'s study were examined pairwise to -find the -frequency with which 
the two criteria are met. O-f the 5 sets o-f autocorrel ated data, 4 met the two 
criteria o-f similarity. Speci-f ical 1y, the 3rd and 7th and the 4th and 6th 
observers o-f the "questioning" behavior produced two pairs o-f data sets that 
met the two criteria. (Note: had a less stringent criteria o-f 95X con-fidence 
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interval instead o-f the 68% interval been used, all -four sets were within 2 
standard errors o-f one another.) All 4 data sets can be described as 
SARIMA(9,0,0) Riodels with cycles o-f 9 data points. The estimated SAR(9) 
parameters are .217 (68XCI: .128-. 306) -for the 3rd observer, .266 (68'/<C!: 
.178-. 354) ^or the 7th observer, and .376 (68%CI : .290-. 462) ^or the 4th 
observer, and .363 (68%CI : .274-. 452) -for the 6th observer. In other words, 
while all SAR(9) parameter estimates were well within the 957.01 o-f one another, 
the 3rd ^nd 7th observers and the 4th and 6th observers -formed distinct pairs 
o-f particularly similar parameters. Since 4 out o-f 5 au tocorrel ated series met 
the two conditions speci-fied above -for the application o-f the Suen-Lee method 
with negligible in-flation o-f variances o-f variance estimates, one may conclude 
that restricting the use o-f the Suen-Lee method to situations in which the two 
criteria are met may not seriously lessen its applicability. 



THIRD ANALYSIS 



To investigate whether unexpected problems may arise in the application o-f 
the Suen-Lee method when the two criteria o-f similarity are met, the data -from 
the 3rd and 7th and those -from the 4th and 6th observers were submitted to two 
separate analyses through the Suen-Lee method. To determine the e-f-ficacy o-f 
the Suen-Lee method used under the restriction o-f similar ARIMA or SARIMA 
parameters, the variance estimates be-fore and a-fter -f i 1 ter i ng were compared. 
Based on the model in Eq . (2), the variance estimates o-f the filtered data 
should be larger than their un-filtered data counterparts. In other words, if 
any the filtered variance estimates is smaller than its unfiltered counterpart, 
either the process is not effective or the model in Eq. is misspecif ied. 
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DATA ANALYSIS AND RESULTS 

The Suen-Lee method was applied to the two pairs o-f data sets and the 
back-forecasting method was used to -filter autocorrelation -from the data. For 
the purpose o-f -filtering, the SARIMA(9 ,0 ,0) parameters o-f the mean series were 
used. Speci-f ical 1y, the 0 estimate -for the mean series between the 3rd and 7th 
observers was .2805 (std. dev.=.0885, t=3.17, p<.05) and it was .4036 (std. 
dev. = .0862, t=4.68, p<.05) -for the mean series between the 4th and 6th 
observers. The -filtered data were then submitted to a random-e-f-f ect two-way 
ANOWA process in the usual -fashion. The variance estimates were obtained 
through the VARCOMP procedure in SAS, as recommended by Bell (1985). Table 1 
presents the subsequent -filtered and un-filtered variance estimates and 
associated general Izabi 1 i ty coe-f-f i c i en ts. 



Insert Table 1 about here 



For both observer pairs, the interaction (or random error) variance 
estimates -for the -filtered data were smaller than those o-f the un-filtered data, 
as hypothesized. For the 3rd-7th observer pair, the interaction variance from 
the un-filtered data shows an underestimate o-f only 1.3X when compared against 
the -filtered variance estimate. For the 4th-6th obsei ver pair, however, the 
interaction variance was underestimated by lOX. Since the SARIMA(9,0 ,0) 
parameter estimate -for the 3rd-7th pair was also smaller than that -for the 
4th-6th pair (i.e., .2805 vs. .4036), it provides limited support to the belie-f 
that the biased e-f-fects o-f autocorrelation on interaction variance estimate is 
a direct -function o-f the magnitude o-f the autoregressi ve parameter (e.g., Tiao 
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& Tan, 1966). 

The systematic observer error variance (i.e., 0^^) estimates remained 
unchanged be-fore and a-fter -filtering -for both observer pairs. In all cases, 
the relative magnitudes oy the est imates were extremely low <i.e., .00001 and 
.0000). The -fact that observer variance remains unchanged is consistent with 
the Suen-Lee model specified in Eq. <2). With the wisdom of hindsight, 
however, the extremely small variance across observers should also have been 
predicted because o-f the criteria used. Given the criteria that the Suen-Lee 
method be applied only when the ARIMA or SARIhW parameters are similar, the 
systematic components o-f the data ircm di-f-ferent obse*">*ers are by de-finition 
quite similar and that their data di-f-fer only in random or white-noise 
•fluctuations. Hence, the systematic variance across observers are small. The 
implication -for this small variance across observers is that subsequent 
estimates o-f cr i ter i on-re-f erenced general i zabi 1 i ty coe-f-f i c i en ts woul d be very 
similar to their norm-re-f erenced counterparts. In any event, the -findings 
con-firm the hypothesis that autocorrelation will have negligible or no e-f-fect 
on the systematic variance across observers. 

The estimations o-f systematic variance across time (i.e., 0^ ) 
provided mixed results. For the 3rd-7th observer pair, the -filtered 0^ 
estimate was indeed larger than the un-filtered 0^ , as hypothesized. 
However, -for the 4th-6th observer pair, the opposite result was obtained. This 
suggests that either the -filtering process has -failed to correct -for the e-f-fect 

2 

o-f autocorrelation on 0^ or the model as described in Eq. <2) was 
misspeci-f ied. Based on the -fact that previous applications o-f the method 
(i.e., Suen and Lee, 1987a, 1987b) as well as estimates -for other variance 
components in the present study have produced results as predicted, it is most 
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likely that the effect on 0^^ in Eq, <2> <i-e,, -0)^) was misspeci^ied and 
that, in -fact, (0^ can be either positive or negative in value. 

More care-ful logical deduction also supports the notion that (0^ can be 
either positive or negative. The hypothesis that (0^ js negative was based on 
the known characteristic o-f the t-ratio in the presence o-f autocorrelation. 
However, the underestimation o-f variance in the t-ratio re-flects only the 
e-f-fect o-f autocorrelation on random variance, not systemat ic variance. 
Since variance across time is a systematic variance, the e-f-fect o-f 
autocorrelation on this variance cannot be predicted -from the t-ratio 
deduction. Hence, the more appropriate model is: 



0/ = Ot'*«t*0|<'*^k'*^*' 

where CO^ can take on a positive or negative value, and (0^^ will always be 
negative. 

The di -f-ferences between the -filtered and un-filtered common dependability 
coe-f-f icients (i.e., the norm-re-f erenced general i zabi 1 i ty coe-f-f i c i en t , p ; the 
signal/noise ratio, ^(g); the cr i tericn-re-ferenced general izabi 1 i ty 
coe-f-f i c i en t , 2, and the norm-re-f erenced Cronbach alpha) were practically 
inconsequential -for the 3rd-7th pair, -for which the di -f-ferences were noticeable 
only at the 3rd decimal place. The di -f-ferences were, however, more pronounced 
-for the 4th-6th observer pair* This may suggest again that the e-f-fect o-f 
autocorrelation on the results o-f a general i zabi 1 i ty analysis is a direct 
-function o-f the magnitude o-f the ARIMA parameter. 
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Based on a sample o-f 28 behavioral time series, it was -found in the -first 



none o-f these could be described as an ARIMAd ,0 ,0)) model. When combined with 
the necessity to calculate a Gamma -function within the Tiao-Tan method, this 
suggests that the Tiao-Tan method may have severely limited applied value -for 
the genera] izabi 1 i ty assessment o-f behavicral observation data. 

The Suen-Lee method appears to be a viable alternative provided that the 
in-flation o-f the variances o-f variance estimates can be minimized. One method 
to attain this is to use the Suen-Lee method only when the ARIMA parameters -for 
the data -from di-f-ferent observers are su-f-f iciently similar. Using the criteria 
o-f identical ARIMA or SARIMA models across observers and the parameter 
estimates being within one standard l .or o-f one another, it was -found that 4 
out o-f 5 autocorrelated series could be considered su-f-f i c i en t1 y similar such 
that the application o-f the Suen-Lee method will lead to relatively stable 
variance estimates. 

An implied result o-f limiting the application o-f the Suen-Lee method to 
situations in which the ARIMA parameters across observers are similar is that 
the Suen-Lee method is applicable only to situations in which the relative 
systematic observer variance is small. For these situations, the di -f-ferences 
between estimates -for a norm-re-f erenced vs. a cr i ter i on-re-ferenced 
interpretation becomes negligible. 

It was also -found that the e-f-fect o-f autocorrelation on the systematic 
variance across time is not always negative as previously hypothesized. To 
summarize, in the presence o-f autocorrelation, the systematic variance across 



analysis '<hat 5 < i .e 



18X) contained some -form o-f autocorrelation. However, 
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time may be over- or under-estimated. Autocorrelation along time will have no 
effect on the systematic variance across observers. The interaction or random 
error variance will always be underestimated in the presence of 
autocorrelation. 

A limitation of the above analyses is that they were based on only 28 
behavioral observation time series, despite the fact that 20,160 original 
empirical data points were involved. Given that there were only 28 time 
series, the results may not be representative of all behavioral time series. 
Hence, the conclusions of these analyses can only be considered tentative. 

To investigate the representativeness of the findings of the analyses in 
this paper, two other sets of behavioral data are currently being analyzed. 
Specifically, 25 time series in Brulle and Repp's (1984) study representing 5 
classroom behaviors of a child over 5 days with a total of 44,000 observation 
points and 32 time series in Brusca's (1986) study representing 2 stereotypic 
behaviors of an autistic child over 16 observation sessions are being analyzed. 
Preliminary evidence suggests tf^at the existence of ARIMA<1,0,0> series in 
these data may also be rare and that seasonal models are common, particularly 
among stereotypic behaviors. 
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Table 1 



VARIANCE ESTIMATES OF UNFILTERED AUTOCORREI^TED BEHAVIORAL DATA 
AND THE SAME DATA FILTERED THROUGH THE SUEN-LEE METHOD 



Observers 


Parameter 


Un-f i 1 tered Est imate 


Fi 1 tered E^Jtimate 


3rd & 7th 


X 


.00749 


.00787 




K 


.00001 


.00001 




^ 2 

tK 


.00235 


.00238 






.76112 


.76780 




V<9) 


3.18723 


3.30672 




5 


.76041 


.76706 




Cronbach a 


.86440 


.86865 


4th & «5th 




.00592 


.00530 




A 2 


.00000 


.00000 




5 2 


.00151 


,00167 






.79677 


.76040 






3.92053 


3.17365 




i 


.79677 


.76065 




Cronbach a 


.88689 


.86390 
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